The bootstrap

author: An introduction to resampling methods autosize: false font-family: ‘Gill Sans’ transition: none

Reference: Data Science Chapter 5

Quantifying uncertainty

From the New England Journal of Medicine in 2006:

We randomly assigned patients with resectable adenocarcinoma of the stomach, esophagogastric junction, or lower esophagus to either perioperative chemotherapy and surgery (250 patients) or surgery alone (253 patients)…. With a median follow-up of four years, 149 patients in the perioperative-chemotherapy group and 170 in the surgery group had died. As compared with the surgery group, the perioperative-chemotherapy group had a higher likelihood of overall survival (five-year survival rate, 36 percent vs. 23 percent).

Quantifying uncertainty

Conclusion: - Chemotherapy patients are 13% more likely to survive past 5 years.

Quantifying uncertainty

Conclusion: - Chemotherapy patients are 13% more likely to survive past 5 years.

Not so fast! In statistics, we ask “what if?” a lot:
- What if the randomization of patients just happened, by chance, to assign more of the healthier patients to the chemo group?
- Or what if the physicians running the trial had enrolled a different sample of patients from the same clinical population?

Quantifying uncertainty

Conclusion: - Chemotherapy patients are 13% more likely to survive past 5 years.

Always remember two basic facts about samples:
- All numbers are wrong: any quantity derived from a sample is just a guess of the corresponding population-level quantity.
- A guess is useless without an error bar: an estimate of how wrong we expect the guess to be.

Quantifying uncertainty

Conclusion: - Chemotherapy patients are 13% \(\pm\) ? more likely to survive past 5 years, with ??% confidence.

By “quantifying uncertainty,” we mean filling in the blanks.

Quantifying uncertainty

In stats, we equate trustworthiness with stability: - If our data had been different merely due to chance, would our answer have been different, too? - Or would the answer have been stable, even with different data?

\[ \begin{array}{r} \mbox{Confidence in} \\ \mbox{your estimates} \\ \end{array} \iff \begin{array}{l} \mbox{Stability of those estimates} \\ \mbox{under the influence of chance} \\ \end{array} \]

Quantifying uncertainty

For example: - If doctors had taken a different sample of 503 cancer patients and gotten a drastically different estimate of the new treatment’s effect, then the original estimate isn’t very trustworthy. - If, on the other hand, pretty much any sample of 503 patients would have led to the same estimates, then their answer for this particular subset of 503 is probably accurate.

Let’s work through a thought experiment…

Kolmorogov goes fishing…

Kolmorogov goes fishing…

Imagine Andrey Kolmorogov on four-day fishing trip. - The lake is home to a very large population of fish of varying size and weight.
- On each day, Kolmorogov takes a random sample of size \(N=15\) from this population—that is, he catches (and releases) 15 fish.
- He records the weight and approximate volume of each fish.
- He uses each day’s catch to compute a different estimate of the volume–weight relationship for all fish in the lake.

Kolmorogov goes fishing…

Kolmorogov goes fishing…

Kolmorogov goes fishing…

At right we see the sampling distribution for both \(\beta_0\) and \(\beta_1\).
- Each is centered on the true population value.
- The spread of each histogram tells us how variable our estimates are from one sample to the next.

Some notation

Suppose we are trying to estimate some population-level quantity \(\theta\): the parameter of interest.

So we take a sample from the population: \(X_1, X_2, \ldots, X_N\).

We use the data to form an estimate \(\hat{\theta}_N\) of the parameter.

Some notation

Suppose we are trying to estimate some population-level quantity \(\theta\): the parameter of interest.

So we take a sample from the population: \(X_1, X_2, \ldots, X_N\).

We use the data to form an estimate \(\hat{\theta}_N\) of the parameter.

Now imagine repeating this process thousands of times!

Some notation

Key definitions

Estimator: any method for estimating the value of a parameter (e.g. sample mean, sample proportion, slope of OLS line, etc).

Sampling distribution: the probability distribution of an estimator \(\hat{\theta}_N\) under repeated samples of size \(N\).

Bias: Let \(\bar{\theta}_N = E(\hat{\theta}_N)\) be the mean of the sampling distribution. The bias of \(\hat{\theta}_N\) is \((\bar{\theta}_N - \theta)\): the difference between the average answer and the truth.

Unbiased estimator: \((\bar{\theta}_N - \theta) = 0\).

Standard error:

Standard error: the standard deviation of an estimator’s sampling distribution

\[ \begin{aligned} \mbox{se}(\hat{\theta}_N) &= \sqrt{ \mbox{var}(\hat{\theta}_N) } \\ &= \sqrt{ E[ (\hat{\theta}_N - \bar{\theta}_N )^2] } \\ &= \mbox{Typical deviation of $\hat{\theta}_N$ from its average} \end{aligned} \]

“If I were to take repeated samples from the population and use this estimator for every sample, how much does the answer vary, on average?”

Standard error

If an estimator is unbiased, then

\[ \begin{aligned} \mbox{se}(\hat{\theta}_N) &= \sqrt{ E[ (\hat{\theta}_N - \bar{\theta}_N )^2] } \\ &= \sqrt{ E[ (\hat{\theta}_N - \theta )^2] } \\ &= \mbox{Typical deviation of $\hat{\theta}_N$ from the truth} \end{aligned} \]

“If I were to take repeated samples from the population and use this estimator for every sample, how big of an error do I make, on average?”

An analogy

An analogy

This is why doctors and lawyers are buying “farmhouses.”

An analogy

An analogy

Chip and Joanna lifestyle item #1: the farmhouse sink

The farmhouse idyll…

And the fine print

Manufacturing tolerances

• On average across many weeks of manufacturing, the fancy sink has width equal to 30".
  
• But individual sinks vary from the average by about 0.5", due to manufacturing variability.
  
• So I expect that my specific sink will be somewhere in the vicinity of 30" \(\pm\) 0.5".

Don’t make any lifestyle choices that require greater precision!

Standard errors

• On average across many samples, my estimator \(\hat{\theta}_N\) is equal to the right answer (\(\theta\)).
  
• But individual estimates vary from the average by about \(\mbox{se}(\hat{\theta}_N)\), due to sampling variability.
  
• So I expect that the right answer is somewhere in the vicinity of \(\hat{\theta}_N \pm \mbox{se}(\hat{\theta}_N)\).

Don’t reach any scientific conclusions that require greater precision!

Standard errors

But there’s a problem here… - Knowing the standard error requires knowing what happens across many separate samples.
- But we’ve only got our one sample!
- So how can we ever calculate the standard error?

Standard errors

Two roads diverged in a yellow wood
And sorry I could not travel both
And be one traveler, long I stood
And looked down one as far as I could
To where it bent in the undergrowth…

–Robert Frost, The Road Not Taken, 1916

Quantifying our uncertainty would seem to require knowing all the roads not taken—an impossible task.

The bootstrap

Problem: we can’t take repeated samples of size \(N\) from the population, to see how our estimate changes across samples.

Seemingly hacky solution: take repeated samples of size \(N\), with replacement, from the sample itself, and see how our estimate changes across samples. This is something we can easily simulate on a computer.

Basically, we pretend that our sample is the whole population and we charge ahead! This is called bootstrap resampling, or just bootstrapping.

Sampling with replacement is key!

Bootstrapped sample 1

Sampling with replacement is key!

Bootstrapped sample 2

Sampling with replacement is key!

Bootstrapped sample 3

The true sampling distribution

The bootstrapped sampling distribution

The bootstrapped sampling distribution

• Each bootstrapped sample has its own pattern of duplicates and omissions from the original sample.
  
• These duplicates and omissions create variability in \(\hat{\theta}\) from one bootstrapped sample to the next.
  
• This variability mimics the true sampling variability you’d expect to see across real repeated samples from the population.

Bootstrapping: pseudo-code

• Start with your original sample \(S = \{X_1, \ldots, X_N\}\) and original estimate \(\hat{\theta}_N\).

• For \(b=1,...,B\):

  1. Take a bootstrapped sample \(S^{(b)} = \{ X_1^{(b)}, \ldots, X_N^{(b)} \}\)
     
  2. Use \(S^{(b)}\) to re-form the estimate \(\hat{\theta}_N^{(b)}\).

• Result: a set of \(B\) different estimates \(\hat{\theta}_N^{(1)}, \hat{\theta}_N^{(b)}, \ldots, \hat{\theta}_N^{(B)}\) that approximate the sampling distribution of \(\hat{\theta}_N\).

Then what?

Calculate the bootstrapped standard error as the standard deviation of the bootstrapped estimates:

\[ \hat{se}(\hat{\theta}_N) = \mbox{std dev}\left( \hat{\theta}_N^{(1)}, \hat{\theta}_N^{(b)}, \ldots, \hat{\theta}_N^{(B)} \right) \]

Or: form an interval estimate based on the the quantiles (e.g. the 2.5 and 97.5 percentiles) of the bootstrapped estimates. This is called a bootstrapped confidence inverval.

Example

type: prompt

Let’s dig in to some R code and data: creatinine_bootstrap.R and creatinine.csv (both on class website).

We’ll bootstrap two estimators:
- the sample mean
- the OLS estimate of a slope

Confidence intervals

Informally, an interval estimate is a range of plausible values for the parameter of interest.

We’d like to be able to asociate

Confidence intervals

If an interval estimate satisfies the frequentist coverage principle, we call it a confidence interval:

Frequentist coverage principle: If you were to analyze one data set after another for the rest of your life, and you were to quote X% confidence intervals for every estimate you made, those intervals should cover their corresponding true values at least X% of the time. Here X can be any number between 0 and 100.

Confidence intervals